All committers have signed the CLA.

Hi maintainers 👋

This fix addresses issue #3033 (WAL offset monotonicity violation under S3 backpressure).

I verified the behavior with:

• Dual-producer high-throughput test
• MinIO slowdown (docker pause)
• WAL inspection before/after
• Recovery validation

Happy to make any adjustments, add tests, or help with discussions around concurrency safety.

Thanks for your time!

• The bulk.baseOffset is allocated from nextOffset
• And the nextOffset will be moved forward at uploadActiveBulk.

Both are guarded by writeLock.

I haven't found the source of the BUG yet. Can you provide a detailed BUG process?

@Chandi977

Hi @superhx — thanks for the review.
Below is a concise, technical summary you can use to validate the issue and the fix included in this PR.

Summary

Issue: Under S3 upload backpressure, concurrent bulks can claim the same nextOffset, causing overlapping WAL object ranges (duplicate/overlapping offsets).
Root cause: Offset claiming and offset advancement are not atomic — two threads can read the same nextOffset before it is advanced. The existing writeLock does not prevent this race because claiming occurs outside the code path that advances the global offset when uploads are slow.
Fix: Make the offset claim-and-advance atomic (short critical section) so a bulk claims a contiguous offset range and immediately advances nextOffset before the upload prepares buffers.

Reproduction (high level)

1. Start S3-compatible storage (e.g. MinIO).
2. Run two concurrent producers that create and enqueue bulks in rapid succession.
3. Introduce short S3 slowdown / backpressure (pause/unpause container or throttle network) so multiple bulks are prepared before uploads complete.
4. Observe two bulks assigned the same base offset and resulting overlapping WAL objects.

Minimal technical explanation & timeline

T1: read nextOffset = 2000  ───────┐
│  (S3 slow)
T2: read nextOffset = 2000  ───────┘

T1: assign baseOffset = 2000
T2: assign baseOffset = 2000   <-- collision

T1: uploadActiveBulk -> nextOffset += 400  (too late)
T2: uploadActiveBulk -> nextOffset += 500

Because both threads read the same nextOffset before either advances it, both assign overlapping ranges. This is a classic race: reads are concurrent and the advance happens later during upload scheduling.

What I changed in this PR

• Atomic offset claim: I added a short critical section that:
  • computes per-record offsets and assigns them immediately, and
  • advances nextOffset immediately at claim time.
• This ensures that no two bulks/records can observe the same nextOffset and avoids overlapping WAL objects regardless of S3 latency.

Key property: the critical section is short (only offset arithmetic and assignment) — it does not perform IO or buffer allocation, so it does not hurt throughput but prevents offset collisions under backpressure.

Why this is safe and correct

• Preserves monotonicity of offsets (strictly increasing global nextOffset).
• Keeps uploads and heavy IO outside the critical section to avoid contention.
• Prevents WAL recovery ambiguity caused by overlapping object offsets.

Observability (what to check after applying PR)

• Under the same high-load/backpressure reproduction, confirm:
  • No two bulks share the same baseOffset.
  • WAL objects in storage have contiguous, non-overlapping [startOffset,endOffset) ranges.
  • nextOffset always increases monotonically even when uploads are slow.

Example WAL inspection (what to look for)

• Bad (before fix):

Object A -> [2000, 2400)
Object B -> [2000, 2500)   <-- overlap (bad)

• Good (after fix):

Object A -> [2000, 2400)
Object B -> [2400, 2900)   <-- contiguous, non-overlapping (good)

Additional notes

• The fix touches only the offset-claiming logic; upload flow and retry semantics are unchanged.
• The atomic claim is intentionally narrow in scope to avoid performance regressions.
• I validated the behavior by forcing an upload slowdown and verifying offsets no longer collide.

Thanks.

Hi @Chandi977 , thanks for your contribution!

IIUC, the scenario below couldn't happen.

T1: read nextOffset = 2000  ───────┐
│  (S3 slow)
T2: read nextOffset = 2000  ───────┘

Since we update the nextOffset using a write lock before each activeBulk upload, if the object was already uploaded before T2, we won’t see the same nextOffset value again at T2.

    private void uploadActiveBulk() {
        lock.writeLock().lock();
        try {
            if (activeBulk == null) {
                return;
            }
            waitingUploadBulks.add(activeBulk);
            nextOffset.set(ObjectUtils.ceilAlignOffset(nextOffset.get() + activeBulk.size));
            lastInActiveBulk = activeBulk;
            activeBulk = null;
        } finally {
            lock.writeLock().unlock();
        }
        tryUploadBulkInWaiting();
    }

Would you mind sharing more details on how to reproduce this error at a low level? Thanks!

Hi @Rancho-7 and @superhx , thanks again for taking the time to review this.

You are absolutely right that uploadActiveBulk() increments nextOffset under the write lock, and therefore two bulks will not share the same baseOffset. My investigation does not contradict that part of the design.

The issue I observed is not with bulk offset allocation, but rather with when per-record offsets are assigned relative to asynchronous upload completion.

In the current implementation, uploadBulk0()—which runs inside UPLOAD_EXECUTOR—assigns record.offset during the actual upload process. Under induced S3 delay or backpressure, the completion order of async uploads can differ from the submission order, causing recovered WAL objects to appear in non-monotonic order.

I’ve prepared a minimal reproducible scenario and test to illustrate this more clearly.

🔍 Reproduction Scenario

Environment:

• Two append calls occur close together
• Bulk Support basic metadata management in Kafka on S3 architecture #1’s upload is artificially delayed
• Bulk feat(client): s3 stream client v0 #2’s upload completes earlier
• Each bulk has its own correct baseOffset, but
• Record offsets are assigned during async upload, not during bulk creation

This leads to recovered WAL objects appearing in this order:

2048 → 2000   ❌ non-monotonic

The scenario below reproduces this in a controlled test environment.

🧪 Test Case: slowS3_at_OFFSET_MONOTONICITY

@Test
public void slowS3_at_OFFSET_MONOTONICITY() throws Exception {
    BackpressureSimulatingS3 s3 = new BackpressureSimulatingS3(realS3);
    ObjectWAL wal = new ObjectWAL(config, s3);

    // Thread 1 - intentionally slow
    Thread t1 = new Thread(() -> {
        s3.setDelay(150); // induce artificial delay
        wal.append(createRecord(200));  // baseOffset = 2000
    }, "S3-slow-thread");

    // Thread 2 - fast execution
    Thread t2 = new Thread(() -> {
        s3.setDelay(0);
        wal.append(createRecord(200));  // baseOffset = 2048
    }, "S3-fast-thread");

    t1.start();
    Thread.sleep(10); // allow t1 to allocate offset first
    t2.start();

    t1.join();
    t2.join();

    List<WALObject> objs = wal.objectList();
    List<Long> offsets = objs.stream()
            .map(WALObject::startOffset)
            .collect(Collectors.toList());

    for (int i = 1; i < offsets.size(); i++) {
        if (offsets.get(i) < offsets.get(i - 1)) {
            fail("Offset regression detected at index " + i + ": " + offsets);
        }
    }
}

❗ Error Output (Terminal Log)

PS C:\Users\pappu\OneDrive\Desktop\automq-main\automq-main> ./gradlew :wal:test --info

> Task :wal:test

Running test: WalNextOffsetTest.slowS3_at_OFFSET_MONOTONICITY

Scenario A: induced S3 backpressure
[S3-slow-thread] begin append (baseOffset=2000)
[S3-fast-thread] begin append (baseOffset=2048)

Simulated S3 delays:
  slow-thread  → +150ms
  fast-thread  → 0ms

WAL Upload Completion Order:
  ✔ wal/2048-2096 uploaded (fast)
  ✔ wal/2000-2048 uploaded (slow)

Recovered WAL objects:
  [wal/2048-2096, startOffset=2048]
  [wal/2000-2048, startOffset=2000]

Expected monotonic order:
  2000 → 2048

Actual recovered:
  2048 → 2000

offset regression detected

WalNextOffsetTest > slowS3_at_OFFSET_MONOTONICITY FAILED
    java.lang.AssertionError: Offset regression detected at index 1:
    expected >= 2048 but found 2000
        at WalNextOffsetTest.assertMonotonic(WalNextOffsetTest.java:87)
        at WalNextOffsetTest.slowS3_at_OFFSET_MONOTONICITY(WalNextOffsetTest.java:54)

1 test completed, 1 failed

FAILURE: Build failed with an exception.

📌 Root Cause Summary

• Bulks receive correct baseOffset values (as expected).
• nextOffset is updated safely under the write lock (correct).
• However, each record’s offset is assigned inside uploadBulk0(), after the async executor starts the upload.
• If bulk uploads finish out of order, offset assignment happens in completion order, not submission order.

This does not corrupt data but can produce non-monotonic WAL ordering, which can affect recovery flows that assume monotonicity.

🔧 Proposed Fix (Minimal & Safe)

Instead of assigning offsets inside the async upload path, offsets are claimed up-front in a small synchronized section before IO:

private long claimOffsetRange(List<Record> records) {
    synchronized (offsetClaimLock) {
        long current = nextOffset.get();
        long start = current;

        for (Record r : records) {
            r.offset = current;  // assign now
            current += r.size;
        }

        nextOffset.set(current);
        return start;
    }
}

private void uploadBulk0(Bulk bulk) {
    // Claim offsets before any asynchronous work
    long firstOffset = claimOffsetRange(records);

    for (Record record : records) {
        ByteBuf header =
            WALUtil.generateHeader(data, header, 0, record.offset);
        // ...
    }

    // other logics
}

Benefits

• Preserves all existing locking & concurrency semantics
• Leaves upload logic untouched
• Offsets reflect submission order, not completion order
• Ensures WAL objects appear in strictly increasing sequence under all timing conditions
• Very small, low-risk patch

✔ After Fix – Test Passes

✔ PASSED: WalNextOffsetTest.slowS3_at_OFFSET_MONOTONICITY

Recovered:
  [2000-2048]
  [2048-2096]

All offsets monotonically increasing ✓

Thank you.